Panoramic imaging technology has been used for merging multiple photographs or digital images to produce a single seamless 360° panoramic view of a particular scene. A single photographic camera is usually employed in such a way that a sequence of image inputs is obtained as the camera is rotated around the focal point of the camera lens causing every two neighboring images to slightly overlap each other. The intensity values from the two neighboring images in the overlap region are weighted and then summed to form a smooth transition. The resultant panorama provides a 2D (two-dimensional) description of the environment.
There is a wide range of potential applications that requires not only intensity panorama but also panoramic three-dimensional (3D) maps associated with the intensity images, that is, a 3D description of the environment. VR technology and e-commerce are example applications where 3D panorama plays a crucial role. Virtual world and virtual objects can be built using the 3D panorama and displayed with the help of VRML (Virtual Reality Modeling Language); see Ames et al., VRML 2.0 Sourcebook, Second Edition, Positioning Shapes, Chapter 5, pp. 63–75.
In order to obtain both intensity and 3D panorama, multiple (more than one) cameras are usually utilized in constructing a panoramic 3D imaging system. There have been systems producing depth panoramic images; see Huang et al., “Panoramic Stereo Imaging System with Automatic Disparity Warping and Seaming”, Graphical Models and Image Processing, Vol. 60, No. 3, May 1998, pp. 196–208. Huang's system utilizes a side-by-side camera system in imitating a human viewer. Another such system is described in commonly-assigned U.S. Pat. No. 6,023,588 issued Feb. 8, 2000 to Ray et al., and entitled “Method and Apparatus for Capturing Panoramic Images with Range Data”. Ray's system displaces the camera vertically such that the line between the rear-nodal points of the cameras is aligned with the rotation axis.
Stereo vision techniques are commonly used in multiple camera systems to recover spatial information of the scene. Such systems yield a 3D range image where the range values may not be defined at every pixel. Imaging systems that are capable of recovering range values at every pixel (full 3D range recovery) are known in the art. For example, Cyberware, Inc. manufactures a system whereby a laser is scanned across a scene. Another method described in U.S. Pat. No. 4,935,616 (and further described in the Sandia Lab News, vol. 46, No. 19, Sep. 16, 1994) provides a scannerless range imaging system using either an amplitude-modulated high-power laser diode or an array of amplitude-modulated light emitting diodes (LEDs) to completely illuminate a target scene. An improved scannerless range imaging system that is capable of yielding color intensity images in addition to the 3D range images is described in commonly-assigned, U.S. patent application Ser. No. 09/572,522, now U.S. Pat. No. 6,349,174, filed May 17, 2000 and entitled “Method and Apparatus for a Color Scannerless Range Imaging System”. As used herein, a scannerless range imaging system will be referred to as a “SRI camera” and such a system is used in producing both intensity and 3D panoramas.
The SRI camera may be mounted to swivel at the nodal point at angular intervals and produce images; moreover, as described in commonly-assigned U.S. Pat. No. 6,118,946, these images may be captured as image bundles that are used to generate intensity and 3D range images. Like the conventional two-dimensional panorama formed by stitching two neighboring intensity images together, the three-dimensional panorama is constructed by stitching neighboring 3D images. However, problems arise when two adjacent 3D images in a sequence are merged. The 3D values of an object point measured by the SRI camera system is defined with respect to the local three-dimensional coordinate system that is fixed relative to the camera optical system. The computed 3D values of an object point in the real world space is a function of the orientation of the camera optical axis.
Because of the nature of the SRI system, there is a further problem that must be addressed when merging two adjacent range images. The SRI system actually yields phase values that describe the phase offset for each pixel relative to one wavelength of the modulated illumination. These phase values are then converted to range values (because the modulation frequency is known). This leads to two types of ambiguity. First, if the objects in the scene differ in distances greater than one wavelength of the modulated illumination, the computed range values will reflect discontinuities where the corresponding phase values transitioned from one cycle to the next. This ambiguity problem can be solved by the method described in commonly-assigned, U.S. patent application Ser. No. 09/449,101, now U.S. Pat. No. 6,288,776, which was filed Nov. 24, 1999 in the names of N. D. Cahill et al. and entitled “Method for Unambiguous Range Detection). Even if the first type of ambiguity is resolved, a second type of ambiguity exists. This ambiguity arises because the phase values returned by the SRI system do not contain any information about absolute distance to the camera. The information captured by the SRI system is only sufficient to generate relative range values, not absolute range values. Therefore, the absolute range values differ by the values computed and returned by the SRI system in the range images by some unknown constant. In general, the unknown constant for a given range image is not the same as the unknown constant for another range image. This presents a problem when attempting to merge/stitch two adjacent range images captured from the SRI system. If the unknown constants are not the same, it will be impossible to continuously merge the two images.
Therefore, two problems emerge. The first problem is that the computed 3D values in a given image are not absolutely known; they are only known relative to the other objects in the same image. Thus, an unknown constant offset must be added to every 3D value in the image. However, the constant offsets in subsequent 3D images may be different, and the difference in offsets must be determined in order to correctly merge the 3D values from neighboring scenes. Even if the first problem is solved, the 3D values of an object point in subsequent images are still dependent on orientation of the camera optical axis for each image. Consequently, distortion appears when a sequence of 3D images is used to describe the shape of an object. For instance, a smooth surface object in the three-dimensional space appears as a fragmented smooth surface object after reconstruction, using the untreated 3D images. Three methods have been shown to address the second problem in panoramic 3D map formation. Each method comprises transforming 3D values into some reference coordinate system. As described in commonly assigned, U.S. patent application Ser. No. 09/383,573, now U.S. Pat. No. 6,507,665, filed Aug. 25, 1999 in the names of Nathan D. Cahill and Shoupu Chen, and entitled “Method For Creating Environment Map Containing Information Extracted From Stereo Image Pairs”, a directional transformation transforms 3D values by projecting points orthographically into a reference plane. As also described in Ser. No. 09/383,573, a perspective transformation transforms 3D values by projecting points to the common nodal axis. As described in commonly assigned, copending U.S. patent application Ser. No. 09/686,610, filed 11 Oct. 2000 in the names of Lawrence A. Ray and Shoupu Chen, and entitled “Method for Three Dimensional Spatial Panorama Formation”, an (X,Y,Z,) transformation transforms 3D values into 3-element vectors describing orthographic range to a reference system.
Even though all of these approaches eliminate the problem of individual range images being defined in different coordinate systems, they are useless in the SRI camera system unless the difference in constant range offsets between subsequent images is determined.